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Abstract 


The OECD Programme for International Student Assessment (PISA) provides a framework in 
which over 80 countries collaborate to build advanced global metrics to assess the knowledge, 
skills and character attributes of the students. The design of assessments poses major conceptual 
and technical challenges, as successful learning. Beyond a sound conceptual foundation, PISA 
needed to fulfil a range of sometimes competing demands. The school administrators, 
policymakers, and teachers need to be able to use this assessment information to determine how 
to create better opportunities for student learning. Assessments should also provide productive 
feedback, at appropriate levels of detail, to fuel improvement and accountability decisions at 
each level of the education systems. In fact, also to assess of reading, mathematics and science 
as key foundation skills, PISA is now progressively incorporating also some of the broader 
cognitive, social and emotional competencies. 
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Resumen 

El Programa para la Evaluacion Internacional de Alumnos (PISA) proporciona un marco en el que 
mas de 80 paises colaboran para construir metricas globales avanzadas para explorar los 
conocimientos, habilidades y atributos del estudiantado. El disefio de las evaluaciones plantea 
importantes desafios conceptuales y tecnicos, como el exito del aprendizaje. Mas alia de una base 
conceptual solida, PISA necesita cumplir una serie de exigencias a veces contradictorios. Los/as 
administradores de escuelas, autoridades normativas y los docentes deben ser capaces de utilizar 
esta informacion de la evaluacion para determinar como crear mejores oportunidades para el 
aprendizaje de los estudiantes. Las evaluaciones tambien deben proporcionar retroalimentacion 
productiva, con niveles apropiados de detalle, para potenciar las decisiones de mejora y rendicion 
de cuentas, en todos los niveles educativos. De hecho, ademas de evaluar la lectura, las 
matematicas y la ciencia como competencias clave, PISA va incorporando progresivamente 
tambien algunas de las competencias cognitivas mas amplias, sociales y emocionales. 
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The demands on modem education systems 
are evolving fast. In the past, education was 
about teaching people something. Now, it’s 
about making sure that students develop a 
reliable compass and the navigation skills to 
find their own way through an increasingly 
uncertain, volatile and ambiguous world. 

These days, we no longer know exactly 
how things will unfold, often we are surprised 


and need to learn from the extraordinary, and 
sometimes we make mistakes along the way. 
And it will often be the mistakes and failures, 
when properly understood, that create the 
context for learning and growth. A generation 
ago, teachers could expect that what they 
taught would last for a lifetime of their 
students. Today, schools need to prepare 
students for more rapid economic and social 
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change than ever before, for jobs that have not 
yet been created, to use technologies that have 
not yet been invented, and to solve social 
problems that we don’t yet kn ow will arise. 

How do we foster motivated, engaged 
learners who are prepared to conquer the 
unforeseen challenges of tomorrow, not to 
speak of those of today? The dilemma for 
educators is that routine cognitive skills, the 
skills that are easiest to teach and easiest to 
test, are also the skills that are easiest to 
digitize, automate and outsource. There is no 
question that state-of-the-art knowledge and 
skills in a discipline will always remain 
important. Innovative or creative people 
generally have specialized skills in a field of 
knowledge or a practice. And as much as 
‘learning to learn’ skills are important, we 
always learn by learning something. 
However, educational success is no longer 
mainly about reproducing content knowledge, 
but about extrapolating from what we know 
and applying that knowledge in novel 
situations. Put simply, the world no longer 
rewards people just for what they know - 
Weibo or Google knows everything - but for 
what they can do with what they know. 
Because that is the main differentiator today, 
education is becoming more about ways of 
thinking, involving creativity, critical 
thinking, problem-solving and decision- 
making; about ways of working, including 
communication and collaboration; about tools 
for working, including the capacity to 
recognize and exploit the potential of new 
technologies; and, last but not least, about the 
social and emotional skills that help people 
live and work together (Schleicher, 2012). 

Conventionally our approach to problems 
was breaking them down into manageable bits 
and pieces, and then to teach students the 
techniques to solve them. But today we create 
value by synthesizing the disparate bits. This 
is about curiosity, open-mindedness, making 
connections between ideas that previously 
seemed unrelated, which requires being 
familiar with and receptive to knowledge in 
other fields than our own. If we spend our 
whole life in a silo of a single discipline, we 


will not gain the imaginative skills to connect 
the dots where the next invention will come 
from. 

The world is also no longer divided into 
specialists and generalist. Specialists 
generally have deep skills and narrow scope, 
giving them expertise that is recognized by 
peers but not valued outside their domain. 
Generalists have broad scope but shallow 
skills. What counts increasingly are the 
versatilists who are able to apply depth of 
skill to a progressively widening scope of 
situations and experiences, gaining new 
competencies, building relationships, and 
assuming new roles. They are capable not 
only of constantly adapting but also of 
constantly learning and growing, of 
positioning themselves and repositioning 
themselves in a fast changing world. 

Perhaps most importantly, in today’s 
schools, students typically leam individually 
and at the end of the school year, we certify 
their individual achievements. But the more 
interdependent the world becomes, the more 
we rely on great collaborators and 
orchestrators who are able to join others in 
life, work and citizenship. Innovation, too, is 
now rarely the product of individuals working 
in isolation but an outcome of how we 
mobilize, share and link knowledge. Schools 
need to prepare students for a world in which 
many people need to collaborate with people 
of diverse cultural origins, and appreciate 
different ideas, perspectives and values; a 
world in which people need to decide how to 
trust and collaborate across such differences; 
and a world in which their lives will be 
affected by issues that transcend national 
boundaries. Expressed differently, schools 
need to drive a shift from a world where 
knowledge that is stacked up somewhere 
depreciating rapidly in value towards a world 
in which the enriching power of 
communication and collaborative flows is 
increasing (Schleicher, 2015). 

In many schools around the world, 
teachers are trying to help students develop 
such kinds of knowledge, skills and character 
attributes. But education systems are often 
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still struggling with reflecting these in the 
tests and assessments that are used to validate 
what students know and can do. Indeed, the 
vast majority of tasks on many conventional 
tests can now be solved with the help of a 
smartphone. If we want to know whether 
students are smarter than a smartphone, we 
need to build more advanced tests and 
assessments. 

PISA project 

The OECD Programme for International 
Student Assessment (PISA) provides a 
framework in which over 80 countries 
collaborate to build advanced global metrics 
to assess the knowledge, skills and character 
attributes that matter for student success and 
that are essential for full participation in 
modern societies. 

The design of assessments poses major 
conceptual and technical challenges, as 
successful learning is as much about the 
process as it is about facts and figures. The 
design of PISA began with the establishment 
of a conceptual framework which extended 
from: the development of a working 
definition for the assessment areas to be 
assessed and the description of the 
assumptions that underlay that definition; an 
examination of how to organize sets of tasks 
constructed in order to report to policy- 
makers and researchers on performance in 
each assessment area; the identification of a 
set of key characteristics to be taken into 
account when assessment tasks are 
constructed in ways that meaningfully reflect 
learning progressions; the operationalization 
of the set of key characteristics to be used in 
test construction; and the validation of the 
variables, and assessment of the contribution 
which each made to the understanding of task 
difficulty. The framework then provided the 
foundation for the design of tasks that can be 
used to generate informative student 
responses, the coding/valuing of those 
responses, the delivery of the tasks and the 
gathering of the responses, and the modelling 
of the responses with respect to the constructs 
to be assessed. 


Beyond a sound conceptual foundation, 
PISA needed to fulfil a range of sometimes 
competing demands. Most obviously, 
assessments need to be fair, technically sound 
and valid for purpose. They also need to build 
on a range of methods to ensure adequate 
measurement of intended constructs and 
measures of different grain size to serve 
decision-making needs at different levels of 
the education system. Assessments should 
also provide productive feedback, at 
appropriate levels of detail, to fuel 
improvement and accountability decisions at 
each level of the education systems. Teachers 
need to be able to understand what the 
assessment reveals about students’ thinking. 
And school administrators, policymakers, and 
teachers need to be able to use this assessment 
information to determine how to create better 
opportunities for student learning. PISA faces 
the added challenge of ensuring that the 
outcomes are valid across the cultural, 
national and linguistic boundaries over which 
they extend and that the target populations 
from which the samples in the participating 
countries are drawn are comparable 
(Schleicher, 2014). 

PISA began its assessments in 2000 with 
an in-depth assessment of reading literacy 
skills, defined as understanding, using, 
reflecting on and engaging with written texts. 
This concept deliberately went beyond the 
traditional notion of decoding information and 
literal interpretation of what is written, and 
encompassed a range of situations in which 
people read, the different ways written texts 
are presented through different media, and the 
variety of ways that readers approach and use 
texts, from the functional and finite, such as 
finding a particular piece of practical 
information, to the deep and far-reaching, 
such as understanding other ways of doing, 
thinking and being. Since 2000, the PISA 
reading assessments have evolved 
considerably, reflecting in large parts the 
changes in the nature of reading with the 
advent of digital technologies. In the past, 
teachers could tell students to look 
information they were missing up in an 
encyclopaedia, and to rely on that infonnation 
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generally being accurate and true. Nowadays, 
digital texts require students to manage non- 
linear information structures, to build their 
own mental representation of information as 
they find their way through hypertext on the 
internet, and to deal with ambiguity and to 
interpret and resolve conflicting information 
which they find somewhere on the web. 
Indeed, the more content knowledge digital 
technologies allow student to search and 
access, the more important becomes the 
capacity to make sense out of this content, 
and the capacity of students to question or 
seek to improve the accepted knowledge and 
practices of their time (OECD, 1999). 

In 2003, the focus of PISA turned to 
assessing mathematics, defined as students’ 
capacity to formulate, employ and interpret 
mathematics in a variety of contexts. The 
assessment was about reasoning 
mathematically and using mathematical 
concepts, procedures, facts, and tools to 
describe, explain and predict phenomena. 
Students were also asked to demonstrate that 
they could recognise the role that 
mathematics plays in the world and to make 
the well-founded judgements and decisions 
needed by constructive, engaged and 
reflective citizens. This approach asserts the 
importance of mathematics for full 
participation in society and it stipulates that 
this importance arises from the way in which 
mathematics can be used to describe, explain 
and predict phenomena of many types. PISA 
has established a set of seven fundamental 
mathematical capabilities that underpin 
performance in the PISA mathematics 
assessments: (1) Communication is both 
receptive and expressive. Reading, decoding 
and interpreting statements, questions, tasks 
or objects enables the individual to form a 
mental model of the situation. Later, the 
problem-solver may need to present or 
explain the solution. (2) Mathematising 
involves moving between the real world and 
the mathematical world. It has two parts: 
formulating and interpreting. Formulating a 
problem as a mathematical problem can 
include structuring, conceptualising, making 
assumptions and/or constructing a model. 


Interpreting involves determining whether 
and how the results of mathematical work are 
related to the original problem and judging 
their adequacy. (3) Representation entails 
selecting, interpreting, translating between 
and using a variety of representations to 
capture a situation, interact with a problem, or 
present one’s work. (4) Reasoning and 
argument is required throughout the different 
stages and activities associated with 
mathematical literacy. This capability 
involves thought processes rooted in logic 
that explore and link problem elements so as 
to be able to make inferences from them, 
check a justification that is given, or provide a 
justification of statements or solutions to 
problems. (5) Devising strategies for solving 
problems is characterised as selecting or 
devising a plan or strategy to use mathematics 
to solve problems arising from a task or 
context, and guiding and monitoring its 
implementation. (6) Using symbolic, formal 
and technical language and operations 
involves understanding, interpreting, 
manipulating and making use of symbolic and 
arithmetic expressions and operations, using 
formal constructs based on definitions, rules 
and formal systems, and using algorithms 
with these entities. Finally, (7) using 
mathematical tools involves knowing about 
and being able to use various tools (physical 
or digital) that may assist mathematical 
activity, and knowing about the limitations of 
such tools (OECD, 2003). 

In 2006, the focus of PISA shifted to 
science, defined as the ability of students to 
engage with science-related issues, and with 
the ideas of science, as a reflective citizen. To 
do well on the PISA science test, students 
need to be willing to engage in reasoned 
discourse about science and technology. This 
requires the competencies to (1) explain 
phenomena scientifically (which implies the 
ability to recognise, offer and evaluate 
explanations for a range of natural and 
technological phenomena), (2) evaluate and 
design scientific enquiry (which implies the 
ability to describe and appraise scientific 
investigations and propose ways of addressing 
questions scientifically) and (3) to interpret 
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data and evidence scientifically (which 
implies the ability to analyse and evaluate 
data, claims and arguments in a variety of 
representations and draw appropriate 
scientific conclusion). Explaining scientific 
and technological phenomena demands a 
knowledge of the content of science. The 
second and third competencies, however, 
require more than a knowledge of what we 
know. Rather, they depend on an 
understanding of how scientific knowledge is 
established and the degree of confidence with 
which it is held. Recognising and identifying 
the features that characterise scientific 
enquiry requires a knowledge of the 
procedures that are the foundation of the 
diverse methods and practices used to 
establish scientific knowledge - referred to 
here as procedural knowledge. Finally, the 
competencies require epistemic knowledge - 
an understanding of the rationale for the 
common practices of scientific enquiry, the 
status of the knowledge claims that are 
generated, and the meaning of foundational 
terms such as theory, hypothesis and data 
(OECD, 2006). 

Challenges for PISA project 

While continuing with the assessment of 
reading, mathematics and science as key 
foundation skills, PISA is now progressively 
incorporating also some of the broader 
cognitive, social and emotional competencies 
discussed above. The assessment of social 
competencies became a priority in 2015. As 
noted before, young individuals entering into 
the workforce and public life need the skills 
and attitudes to collaborate and effectively 
solve problems, increasingly in situations 
where members of the group are 
geographically dispersed, working in different 
time zones, and connected through 
technology. Societies expect them to have the 
capacity to resolve problems and provide 
solutions collaboratively through the pooling 
of knowledge, skills, and effort. As a first 
step, PISA introduced in 2015 an assessment 
of collaborative problem solving skills, which 
assesses students according to three core 
competencies: (1) establishing and 


maintaining shared understanding; (2) taking 
appropriate actions to solve problems; and (3) 
establishing and maintaining team 
organisation. To facilitate this, individual 
students are required to interact and 
collaborate with computer-generated team 
member(s) in controlled situations to solve a 
particular problem. This process necessitates 
students determining their own role and 
responsibilities in regards to other agents, 
monitoring aspects of group organisation, and 
facilitating adjustments and changes that are 
needed when communication breaks down, 
when new obstacles appear, or when 
opportunities for performance optimisation 
arise. 

Taking this further, countries are currently 
collaborating to develop an assessment of 
global competence to be delivered in 2018 
and defined as the capability and disposition 
to act and interact appropriately and 
effectively, both individually and 
collaboratively, when participating in an 
interconnected, interdependent and diverse 
world. The assessment is built around four 
key dimensions: (1) Communication and 
relationship management - which refers to the 
willingness and capability to adapt one’s 
communication and behaviour in order to 
interact appropriately and effectively with 
others holding diverse perspectives and in 
different contexts. (2) Knowledge of and 
interest in global developments, challenges 
and trends - which refers to a learner’s 
interest in and knowledge of cultures, major 
issues, events and phenomena in the world, as 
well as the learner’s ability to understand their 
global significance and their implications for 
adapting appropriately and effectively to 
learning, working, and living situations with 
others holding diverse perspectives and in 
different contexts. (3) Openness and 
flexibility - which refers to being receptive to 
and understanding of new ideas, people and 
situations, as well as to differing perspectives 
and practices. It also refers to the ability to 
seek out and understand new and differing 
perspectives and experiences and 
appropriately and effectively adapt one’s 
thinking, behaviours and actions to learning, 
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working, and living situations that involve 
others holding diverse perspectives and in 
different contexts. (4) Emotional strength and 
resilience - which refers to the ability, 
developing the coping mechanisms and 
resilience, to deal appropriately with the 
ambiguity, changes, and challenges that these 
different perspectives and experiences can 
present. 

As our societies evolve further, the 
countries participating in PISA will continue 
to collaborate to define and measure the 
knowledge, skills and character attributes that 
will help individuals to keep abreast of these 
changes and to meet rising demands. Every 
three years, the results from PISA provide a 
powerful tool that countries can use to 
develop, review and fine-tune their curricular 
standards and education policies with the aim 
to provide the best education possible for all 
of their students. The OECD stands ready to 
support countries in this challenging and 
crucial endeavour. 
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